Computational Image Sensors for On-Sensor-Compression

processing. Real-time image processing system pro­ ceeds in three steps; transducing, read-out and pro­ In this paper, we propose novel image sensors which cessing. The high bandwidth required to transfer data compress image signal. By making use of very fast from the sensor to the processor is also a fundamen­ analog processing on the imager plane, the compres­tal disadvantage of the paradigm because it leads to sion sensor can significantly reduce the amount of pixel high latency when pixel rate gets higher. In addition data output from the sensor. The proposed sensor is in to the disadvantage, the paradigm demands expensive, tended to overcome the communication bottle neck for large computation units in order to perform real-time high pixel rate imaging such as high frame rate imaging application. and high resolution imaging. Tightly combining image acquisition and signal pro­ The compression sensor consists of three parts; cessing is an approach to overcoming the above com­ transducer, memory and processor. Two architectures munication bottfe necks. We have been investigat­ for on-sensor-compression are discussed in this paper ing novel image sensors on which video signal is com­ that are pixel parallel architecture and column parallel pressed using conditional replenishment algorithm [1]. architecture. In the former architecture, the three parts This on-sensor-compression significantly reduces the are put together in each pixel, and processing is pixel image data output from the sensor. Thus, on-sensor­ parallel. In the latter architecture, transducer, proces­compression especially benefits high pixel rate cam­ sor and memory areas are separated, and processing is eras and processing systems. The proposed sensors column parallel. mainly consists of transducer, memory and proces­ We also describe a prototype chip of pixel-parallel­sor. In this paper, we discuss two architectures for type sensor with 32 x 32 pixels which has been fabri on-sensor-compression. They are pixel parallel and col­ cated using 2 J..lm CMOS technology. Some results of umn parallel architecture. examinations are shown in this paper. The pixel-parallel-type sensor has transducer, mem­ ory and processor together in each pixel; processing in pixels is parallel so that processing of the sensor is very fast because it makes use of the two dimen­ 1: Introduction sional nature of image signal. However, its fill factor has to be small by current implementation technology, In order to read out image signal which is inherently and the sensitivity of the sensor is not enough. On two dimensional, usually, image signal is firstly scanned the other hand, the column-parallel-type sensor has into one dimensional signal and read out. When the the transducer area, the memory area, the processor pixel rate gets higher, it becomes more difficult for area separately. The fill factor of the sensor is substan­ conventional image sensors to read out signals. The tially improved because of the separate architecture, bandwidth to transfer data from sensor in the scan­ the power consumption is reduced. A prototype chip and-read-out process is a fundamental limitation for of pixel-parallel-type compression sensor has been de­ very high pixel rate imaging such as very high frame signed and fabricated using 2J..lm CMOS process. Some rate imaging or very high resolution imaging. results of the tests are shown in this paper. Image acquisition is also critical for real-time image Related works in terms of integration of signal pro­ cessing and sensing are found in those areas of ma­ chine vision applications and neural network researches [2][3][4][5]. By integrating processing and sensing, the parallel nature of the image signal can be exploited and the processing gets remarkably faster. In those works, for example, a silicon retina that is a device which computes spatial derivatives of an image and an analog network that calculates optical flow have been developed. Most of those works are focused on how to execute early-vision processing in analog processing. As for imaging devices, CCDs have been dominat­ ing. However, different from CCDs, CMOS-based sen­ sors have been investigated. Recent CMOS based sen­ sors have several amplifier transistors in each pixel. One of those CMOS sensor has a differential opera­ tion mode, that is, it outputs pixel values when the difference between adjacent frames are large[6]. Our proposed sensor is also CMOS-based. One of the essential differences between our sensor and the CMOS sensor[6] is that our proposed sensor has a frame buffer in each pixel. Thus, it can compress current im­ age by using the last reproduction stored in the buffer. On the other hand, the CMOS sensor[6] does not have a frame buffer in each pixel, so it can not detect changes if the difference between adjacent frames is not large enough, and the detection errors can be accumulated. Compared to the conventional digital compression, on-sensor-compression reduces the pixel rate output from the sensor, while the digital compression methods does not. Processing of on-sensor-compression is very fast by making use of parallel nature of image signal. 2: On-Sensor-Compression by Using Conditional Replenishment 2.1: Conditional Replenishment Conditional replenishment [7] is employed for the video compression algorithm on a imager. Conditional replenishment is based on detection and coding of the moving areas so that it makes use of temporal redun­ dancy to compress image signals. As shown in Fig.l, current pixel value is compared to that in the last replenished frame which is stored in the memory. The values and addresses of that pixels are output when the magnitudes of the differences are greater than a threshold which is fixed or controlled by feedback of the number of flag signals. Fig.2 shows the example of the detected moving pix­ els. The resolution of the image is 256 x 240 pixels and compression ratio is 10:1 and 5:1 for threshold 20 and 5 respectively. Although conditional replenishment is rather simple, it can achieve about 10:1 compression Pixel Value In Pixel V31ue Out Refresh Threshold .c. Th ,'~ Figure 1. Description of coding algorithm in each pixel by conditional replenishment Figure 2. Detection of moving pixels top: two frames of an image sequence (256 lev­ els), bottom: pixels detected as moving (left: threshold 5 compression ratio 5:1, right: threshold 20 compression ratio 10:1) ratio without significant degradation under the ordi­ nary circumstance. 2.2: Pixel Rate Control The threshold for rate control is fixed (VPR) or con­ trolled by the number of flag signals (CPR). In case of a fixed threshold, the SNR of the reconstructed image is kept about some value, but the number of activated pixels significantly changes when the scene is changed. The transition of pixel rate is examincd by computer simulation using the image sequcnce which is composed of three uifferent monochrome scquences: Train, Miss. America ,UlU Ra.bbit. Fig.3 shows the cxperimcntal re­ sult when the threshold arc 30 and G, respectively. Trnin Miss America Animal